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(54) Object-oriented image representation method arid apparatus using irregular meshes 



(57) An object-oriented image representation appa- 
ratus using irregular meshes includes an object image 
extractor (10) for extracting image blocks containing an 
object contained in a whole image from the whole image 
represented by received image data. A regular mesh 
former (20) generates rectangular mesh data including 
fixed control points, floating control points and pixel data 
from the extracted object image. The fixed control points 
and the floating control points are vertex points belong- 
ing to an object image among vertex points of a plurality 
of the rectangular meshes which divide the whole image 
and the fixed control points are vertex points located at 



the corners of the object image. An irregular mesh 
former (30) receives the rectangular mesh data and re- 
peats an operation for removing the floating control point 
having image descriptiveness of the lowest degree and 
an operation for dividing a region of support from which 
the floating control point is removed into polygonal 
meshes- until the number of the remaining floating con- 
trol points reaches a predetermined threshold value. 
Thus, since a mesh representation with respect to an 
object instead of a whole image can be done, a much 
more efficient signal processing is possible compared 
with the mesh representation of the whole image. 
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Description 



The present invention relates to an objecloriented image representation method and apparatus using irregular 
meshes, and more particularly, to a method for representing an object contained in an image using-an irregular mesh 

5 and an apparatus thereof. 

An international standardization endeavour with respect to a moving image compression technology, sucn as 
MPEG IV (Moving Picture Experts Group), for coding a moving image at a low bit rate for the purpose of applications 
in a videophone or a video conference system, is in progress. To encode a moving image at a very low bit rate, various 
object-oriented motion image coding methods each of which encodes a significant object in the motion image an« 

w transmits the encoded result are widely under study. Also, to accomplish a motion compensation for removing a tem- 
poral correlation of a motion image, motion compensation methods by which a spatial transformation such as an image 
warping capable ol more effectively representing an actual movement of an object is applied to an image represented 
in the form of a mesh are widely under study. 

In image representation, h can be seen that an irregular mesh can reflect the local features or an image very well 

is and can represent image features well using only a small number of control points. 

With a view to solving or reducing the above problems, it is an aim of preferred embodiments ol the present 
invention to provide a method for representing an object contained in an image using irregular meshes. 

It is another aim of embodiments of the present invention to provide an apparatus embodying the above method. K - 
According to a first aspect of the invention, there is provided an objectnoriented image representation apparatus ?r - 

20 using irregular meshes comprising: object image extraction means for extracting image btocks^containing an object ^ 
contained in a whole image from the whole image represented by received image data; regular mesh formation means & 
for receiving object image data from said object image extraction means and for generating rectangular mesh data «* : 
including fixed control points, floating control points and pixel data from the object image, in which said fixed control ^ 
points and said floating control points are vertex points belonging to an object image among vertex points of a plurality -, 

25 of the rectangular meshes which divide the whole image and said fixed control points are vertex points located at the 
corners of the object image: and irregular mesh formation means for receiving the rectangular mesh data supplied 
from said regular mesh formation means, and repeating an operation for removing said floating control point hav.ng 
image descriptiveness of the lowest degree and an operation for dividing a region of support from which the floating 
control point is removed into polygonal meshes, until the number of the remaining floating control points reaches a 

so predetermined threshold value. * 
Preferably said object image extraction means comprises: block formation means for dividing the whole image of 
the received image data into a plurality of image blocks each having a predetermined size: block selection means for 
selecting image blocks containing an object in the whole image among the plurality of the image blocks divided by said 
block formation means: and object image formation means for merging the image blocks selected by said block se- 

35 lection means and outputting the merged image blocks as object image data. 

Preferably, said irregular mesh formation means performs a division operation for dividing the region of support 
from which the central floating control point has been removed into triangular meshes. 

Said irregular mesh formation means may comprise: regular triangular mesh formation means for determining a 
diagonal direction having a large degree of data similarity of each rectangular mesh using the rectangular mesh data 

40 supplied from' said regular mesh formation means, and forming regular triangular meshes by dividing the rectangular 
meshes along the determined diagonal direction: energy calculation means for calculating energy difference values 
individual corresponding to the floating control points, using the pixel data of the regular triangular meshes formed 
by said regular triangular mesh formation means: control point removal means for determining the smallest energy 
difference value among the calculated energy difference values and removing the floating control point corresponding 

45 toihe determined energy difference value as a non-floating control point: mesh reconstruction means, responsive to 
the floating control point removal operation of said control point removal means, for dividing the region of support from 
which the centre floating control point has been removed into the triangular meshes until the number of the floating 
control points reaches a predetermined threshold value: and energy changing means for calculating the energy differ- 
ence values individually corresponding to the floating control points contained in new triangular meshes using the new 

so triangular meshes and the neighbouring triangular meshes obtained by said mesh reconstruction means, and supplying 
information representing the calculated energy difference values and the corresponding floating control points to satd 

control point removal means. 

Preferably the energy difference value corresponding to each floating control point K represents a difference oe- 
tween a first energy value and a second energy value, wherein the first energy value E(K) involving the floating control 
S5 point K is represented as the following equation. 
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5 

wherein S K represents a region of support corresponding to the floating control point K. Nk represents the numbeir 
of pixels contained in the region of support Sk- gfij) represents a pixel value of the pixel having a position and g 
(ij) represents an interpolated pixel value corresponding to the pixel value gfij) wherein the interpolated pixel value p 

10 (ij) is calculated using the pixel value gfij) and the interpolation weight values corresponding to the pixel value gfij). 
and the interpolation weight values for the pixel having the position (ij) are determined using the pixel values of the 
control points contained in each triangular mesh to which a pixel having the position (ij) belongs within the region of 
support S K before the central floating control point K is removed, and wherein the second energy value with respect 
to the floating control point K is calculated by the above equation using the interpolated pixel value determined by using 

'5 the pixel values of the control points of the new triangular meshes, instead of the interpolated pixel value gfij) for the 
first energy value E(K). - 

Preferably, said irregular rnesh formation means performs a division operation for dividing the region of support 
from which the centre floating control point has been removed into polygonal meshes. 

Said irregular mesh formation means may comprise: energy calculation means for receiving and storing the rec- 
20 tangular mesh data supplied from said regular mesh formation means and calculating and storing energy difference 
Values individually corresponding to the floating control points using the stored rectangular mesh data: and control 
point removal means for repeating a removal operation of a floating control point corresponding to the smallest energy 
" value among the energy values stored in said energy calculation means until the number of the remaining floating 
control points reaches the predetermined threshold value, wherein said energy calculation means newly calculates 
2S and stores the energy values corresponding to the floating control points contained in the region of support from which 
the centre floating control point has been removed. 

Said energy value is preferably calculated by the following equation, 



30 
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35 in which S K represents a region of support corresponding to the floating control point K, N K represents the number 

of pixels contained in the region of support gfij) represents a pixel value of the pixel having a position (ij) and g 
(ij) represents an interpolated pixel value corresponding to the pixel value gfij) wherein the interpolated pixel value g 
(ij) is calculated using the pixel values gfij) and interpolation weight values corresponding to the pixel value gfij), and 
the interpolation weight values for the pixel having the position (ij) are determined using the pixel values of the control 

-w points contained in the region of support S^ 

Preferably, the apparatus further comprises: energy calculation means for calculating energy values involving each 
floating control point and a plurality of candidate control points having a neighbouring pixel position in the region of 
support of each said floating control point, with respect to the floating control points of the polygonal meshes which 
finally remain in said irregular mesh formation means: and means for determining the smallest energy value among 

-ts the energy values calculated by said energy calculation means, and determining a control point corresponding to the 
determined energy value among each said floating control point and the said neighbouring candidate control points 
as an optimal floating control point corresponding to each said floating control point. 
Said energy value is preferably calculated by the following equation. 

so 

5S wherein. S K represents a region of support corresponding to the floating control point K. N K represents the number 

of pixels contained in the region of support Sk. g(ij) represents a pixel value of the pixel having a position (ij) and g 
(ij) represents an interpolated pixel value corresponding to the pixel value gfij), wherein the interpolated pixel value 
gfij) is calculated using the pixel values gfij) and interpolation weight values corresponding to the pixel value gfij) 
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and the interpolation weight values lor the pixel having the position (ij) are determined using the pixel values of the 
control points of the triangular meshes to which the pixel having a position gfl,j) belongs among the triangular meshes 
formed by the floating control point or each said candidate control point and the control points located at the edge erf 
the region of support S*. 

s According to a second aspect of the invention, there is provided an object-oriented image representation method 

using irregular meshes, comprising the steps of: (a) extracting image blocks containing an object contained in a whole 
image from the whole image: (b) g e n e r a t i n g rectangular mesh data including fixed control points, floating control 
points and pixel data from the image blocks extracted in step (a), in which said fixed control points and said floating 
control points are vertex points belonging to an object image among vertex points of a piuraiiiy of the recianguiar 

w meshes which divide the whole image and the fixed control points are vertex points located at the corners of the object 
image: and (c) repeating an operation for removing said floating control point having image descriptiveness of the 
lowest degree and an operation for dividing a region of support from which a centre floating control pint is removed 
into polygonal meshes : until the number of remaining floating control points reaches a predetermined threshold value, 
based on the rectangular mesh data of step (b). 

15 Said step (a) preferably comprises the sub-steps of: (at ) dividing the whole image into a plurality of image blocks 

each having a first predetermined size: (a2) selecting image blocks containing an object in the whole image among 
the plurality of the image blocks of sub-step (at). 

In said step (c). a division operation is preferably performed for dividing the region of support from which the centre 
floating control point has been removed into triangular meshes. 

zo Said step (c) may comprise the sub-steps of: (cl) determining a diagonal direction having a large degree of data 

similarity of each rectangular mesh using the rectangular mesh data of said step (b) and forming regular triangular 
meshes by dividing the rectangular meshes along the determined diagonal direction: (c2) calculating energy difference 
values individually corresponding to the floating control points using the pixel data of the regular triangular meshes 
formed in said sub-step (cl ): (c3) determining the smallest energy difference value among the energy difference values 

2$ calculated in said sub-step (c2) and removing the floating control point corresponding to the determined energy differ- 
ence value as a non-floating control point (c4) dividing the region of support from which the centre floating control 
point has been removed into the triangular meshes in response to the floating control point of said sub-step (c3) when 
the number of the floating control points does not reach a predetermined threshold value: and (c5) calculating the 
energy difference values individually corresponding to the floating control points contained in new triangular meshes 

30 using the new triangular meshes and the neighbouring triangular meshes obtained in said sub-step (c4). 

Preferably, the energy difference value corresponding to each floating control point K represents a difference be- 
tween a first energy value and a second energy value, wherein the first energy value E(K) with respect to the floating 
control point K is represented as the following equation, 

35 _ 



40 

wherein represents a region of support corresponding to the floating control point K ( N K represents the numbejr 
of pixels contained in the region of support S^. g(i,j) represents a pixel value of the pixel having a position (ij). and g 
(ij) represents an interpolated pixel value corresponding to the pixel value g(i.j), wherein the interpolated pixel value 
g(ij) is calculated using the pixel value gfi.j) and the interpolation weight values corresponding to the pixel value g(i, 

*s j), and the interpolation weight values for the pixel having the position fry; are determined using the pixel values of the 
control points contained in each triangular mesh to which a pixel having the position (ij) belongs within the region of 
support S K before the central floating control point K is removed, and wherein the second energy value with respect 
to the floating control point K is calculated by the above equation using the interpolated pixel vaJue determined by using 
the pixel values of the control points of the new triangular meshes instead of the interpolated pixel value g(i.j). 

so Preferably, in said step lc), a division operation is performed for dividing the region of support from which the centre 

floating control point has been removed into rectangular meshes. 

Said step (c) prelerably comprises the sub-steps of: (c1) calculating and storing energy difference values individ- 
ually corresponding to the floating control points using the stored rectangular mesh data of said step (b): and (c2) 
removing the floating control point corresponding to the smallest energy value among the energy values stored in said 

55 sub-step (c1): (c3) calculating the energy values corresponding to the floating control points belonging to the region 
of support from which the centre floating control point has been removed: and (c4) repeating- a removal operation of a 
floating control po*t and a calculation operation of the energy values corresponding to the floating control points 
belonging to the corresponding region of support until the number of the remaining floating control points reaches a 
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predetermined threshold value. 

Said energy value is preferably calculated by the following equation. 



E{ K) = , s Art ». /) - & um 2 



in which S K represents a region of support corresponding to the floating control point K, Nk represents the number 
10 of pixels contained in the region of support Sk, g(ij) represents a pixel value of the pixel having a position (ijh and g 
(i,j) represents an interpolated pixel value corresponding to the pixel value g(ij) t wherein the interpolated pixel value 
gfhj) is calculated using the pixel values gftj) and interpolation weight values corresponding to the pixel value g(ij), 
and the interpolation weight values for the pixel having the position (Lj) are determined using the pixel values of the 
control points contained in the region of support 
is The method may further comprise the steps of: (d) calculating energy values involving each floating control point 

and a plurality of candidate control points having a neighbouring pixel position in the region of support of each said 
floating control point, with respect to the respective floating control points of the polygonal meshes which finally remain 
in step (c); and (e) determining the smallest energy value among the energy values calculated by said step (d) and 
determining a control point corresponding to the determined energy value among each said floating control point and 
20 said neighbouring candidate control points as an optimal floating control point corresponding to each said floating 
control point. 

Said energy value is preferably calculated by the following equation. 



30 . wherein, S K represents a region of support corresponding to the floating control point K. represents the number 

of pixels contained in the region of support S^ g(i'j) represents a pixel value of the pixel having a position (lj) and g 
(ij) represents an interpolated pixel value corresponding to the pixel value g(i,j), wherein the interpolated pixel value 
g(i,j) is calculated using the pixel values g(ij) and interpolation weight values corresponding to the pixel value g(i.j). 
and the interpolation weight values for the pixel having the position (Lj) are determined using the pixel values of the 

3S control points of the triangular meshes to which the pixel having a position g(i.j) belongs among the triangular meshes 
formed by the floating control point or each said candidate control point and the control points located at the edge of 
the region of support S K . 

According toan aspect of the present invention, there is provided an object-orientated image representation method 
using irregular meshes, comprising the steps of: (a) extracting image blocks relating to an object forming part of a 

-to larger, whole image from the whole image: (b) generating rectangular mesh data including fixed control points, floating 
control points and pixel data from the image blocks extracted in step (a), in which the fixed control points and the 
floating control points form selection vertex points which relate to an object image amongst vertex points of a plurality 
of the rectangular meshes which divide the whole image, and in which the fixed control points are those vertex points 
which are located at the comers of the object image: and (c) repeatedly performing an operation for removing a floating 

45 control points having a lowest degree of image descriptiveness and an operation for dividing a region of support from 
which a centre floating control point is removed into polygonal meshes, until the number of remaining floating control 
points reaches a predetermined threshold value, based on the rectangular mesh data of step (b). 

For a better understanding of the invention, and to show how embodiments of the same may be carried into effect, 
reference will now be made, by way of example, to the accompanying diagrammatic drawings, in which: 

so 

Figure 1 is a block diagram of an objectorientated image representation apparatus using irregular .meshes ac- 
cording to one embodiment of the present invention: 

Figures 2A through 2C are conceptual diagrams for explaining a process of forming an object image containing 
55 an object: 

Figure 3 is a conceptual diagram for explaining a meshed object image: 
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Figures 4A through 4C are a conceptual diagram for explaining a process of forming triangular meshes, respec- 
tively: 

Figures 5A and 5B are a conceptual diagram for explaining a removal of a floating control point respectively: 

Figure 6 is a conceptual diagram for explaining a process of deciding an optimal floating control point: and 

Figure 7 is a block diagram showing a unit for forming irregular polygonal meshes according to another embodiment 
of the present invention. 

Preferred embodiments of the present invention will be described with reference to the accompanying Figures 1 
through 7. 

In Figure 1 showing an object-oriented image representation apparatus using irregular meshes according to one 
embodiment of the present invention, an object image extractor 10-extracts an object from an image containing the 

is object. Referring to Figure 2A showing an example of an image containing an object, a block former 11 of the object 
image extractor 10 divides the image shown in Figure 2A comprised of received image data into image blocks of an 
NxN size each image block having NxN pixels. The example of image division by the block former 11 is shown in Figure 
2B. The block former 11 outputs data of the image blocks obtained by the image division to a block selector 12. The 
block selector 12 selects image blocks containing image information of an object among the image blocks using the 

20 received data, the data of the selected image block is output to an object image former 13. In this case, the block 
selector 12 does not output the data of the image blocks exclusive of the selected image blocks to the object image 
former 13. The object image former 1 3 merges the data received from the block selector 12 to thereby form and store 
an object image. The object image is comprised of the image blocks selected by the block selector 12, which is shown 
as an area enclosed by a thick solid line in Figure 2C. The object image former 1 3 outputs the data of the object image 

25 comprised of the image blocks to a rectangular mesh former 20. 

The rectangular mesh former 20 divides a whole image containing an object into grids of a KxK size. Here, each 
grid has a size including KxK pixels. The rectangular mesh former 20 determines vertex points included in the object 
image among the vertex points of each grid as initial control points, and determines each grid enclosed by the initial 
control points as a rectangular mesh. Figure 3 shows an example of the rectangular meshes determined based on the 

30 object image shown in Figure 2C. Trie rectangular meshes shown in Figure 3 represent a case that the number of 
horizontal or vertical pixels of the image block described referring to Figure 2 is an integer multiple of the number of 
the horizontal or vertical pixels of a grid having a KxK size. When the number of the horizontal or vertical pixels of the 
image block is not an integer multiple of the number of the horizontal or vertical pixels of the grid, the object image 
comprised of the rectangular meshes will differ from that comprised of the image blocks. 

3$ The rectangular mesh former 20 determines the initial control points located at the edges of the object image 

comprised of the rectangular meshes as fixed control points, and determines the initial control points which are not 
determined as the fixed control points as floating control points. Here, each rectangular mesh is represented by the 
data of the object image which exists in the rectangular mesh and the position information of a fixed control point or a 
floating control point belonging to the rectangular mesh. The rectangular mesh former 20 outputs the rectangular mesh 

ao data representing the object image comprised of the rectangular meshes. 

An irregular mesh former 30 divides the individual rectangular meshes into irregular triangular meshes using the 
input rectangular mesh data. The irregular triangular meshes are generated based on a paper entitled "Irregular Tri- 
angular Mesh Representation Based on Adaptive Control Point Removal", published in SPIE's 1996 symposium on 
Visual Communications and Image Processing, by Kang W. Chun. Byungwoo Jean and Jae M. Jo. In the irregular 

45 mesh former 30. a regular triangular mesh former 31 receives rectangular mesh data from the rectangular former 20. 
The regular triangular mesh former 31 divides each rectangular mesh in the diagonal direction using the received 
rectangular mesh data. By such division, each rectangular mesh is divided into two regular triangular mesh. 

One example in which the regular triangular mesh former 31 forms the regular triangular meshes will be described 
with reference to Figures 4A through 4C The regular triangular mesh former 31 uses similarity between pixel values 

so belonging to the rectangular mesh to divide each rectangular mesh into the regular triangular meshes. Figure 4 A shows 
a rectangular mesh comprised of four control points a. b. c and d. If the pixel values of the control points a. b. c and d 
belonging to one rectangular mesh are P(a). P(b). P(c) and P(d). respectively, the regular triangular mesh former 31 
calculates a similarity representation value in -45 degree direction using IP(a)-P(c)l. and calculates a similarity 
representation value P 45 in 45 degree direction using IP(b)-P(d)l. The regular triangular mesh former 31 divides the 

55 rectangular mesh into two regular triangular meshes along the diagonal direction corresponding to a smaller value 
among the two similarity representation values P^s and P 45 . Figure 48 shows a division of the rectangular mesh in 
the case when the similarity representation value P^5 is smaller than the similarity representation value P 4S Figure 
4C shows a division of the rectangular mesh in the case when the similarity representation value P^ is smaller than 
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the similarity representation value P^. The regular triangular mesh former 31 generates^^ular triangular mesh data 
representing the regular triangular meshes obtained by the cfivision of all rectangular meshes. Such a regular triangular 
mesh data contains information about the positions of the above described fixed control points and the floating points 
and the position information is supplied to an energy calculator 32. 
5 The energy calculator 32 stores the received regular triangular mesh data in an internal memory (not shown), and 

calculates a first energy value corresponding to each floating control point using the stored data. The first energy value 
E(K) involving the floating control point K is represented as the following equation (1). 



EOO^aT '2 c iu>>] 2 .... (1) 



• 



is Here, represents a region of support corresponding to the floating control point K. and represents the number 

of pixels contained in the region of support S K . g(Lj) represents a pixel value of the pixel having a position fX/J^and g 
(ij) represents an interpolated pixel value corresponding to the pixel value g(ij). The interpolated pixel value g(Lj) is 
calculated using the pixel value g(ij) and the interpolation weight values corresponding to the pixel value g(ij). The 
interpolation weight values for the pixel having the position (ij) are uniquely determined using the pixel values of the 

20 control points-contained in each triangular mesh to which a pixel having the position belongs within the region of 
support S K before the central floating control point K is removed. Referring to Figure 5A, the floating control point 
expressed as "K" has a support region enclosed by the neighbouring eight control points. The energy calculator 32 
calculates the first energy values using the above-described equation (1) with respect to all floating control points in 
the meshed object image, and stores the calculated first energy values in the internal memory in the form corresponding 

25 to the floating control points. The energy calculator 32 removes the floating control point located at the centre of each 
region of support, and divides the region of support from which the floating control point has been removed into triangular 
meshes using a triangulation technique. The triangulation technique is disclosed in the above-described paper of Kang 
W. Chung. Byungwoo Jean and Jae M. Jo. Figure SB shows an example of new triangular meshes which divide the 
region of support from which the floating control point K has been removed. 

so The energy calculator 32 calculates a second energy value using the new triangular meshes contained in the 

region of support from which the central floating control point has been removed. The second energy value involving 
the floating control point K is represented as the above-described equation (1). However, in the calculation of the 
second energy, the interpolated pixel value g(ij) involving the pixel value g(ij) is calculated using pixel values of the 
control points of new triangular meshes. In other words, the interpolation weight values for the pixel having a position 

35 (ij) are uniquely determined using the pixel values of the control points contained in each triangular mesh to which the 
pixel having the position (i.j) belongs. The energy calculator 32 calculates an energy difference value representing a* 
difference between the first energy value and the second energy value corresponding to the first energy value whenever 
the first energy value and the corresponding second energy value are calculated. The signal processing of the energy 
calculator 32 does not restrict the present invention. Thus, the energy calculator 32 can be designed so that energy 

40 values corresponding to the respective floating control points are calculated after the first and second energy difference 
values corresponding to all the floating control points have been calculated. The energy calculator 32 outputs the 
calculated energy difference values to a control point remover 33 in the form corresponding to each floating control 
point. When the energy difference values corresponding to all the floating control points have been completely calcu- 
lated, the energy calculator 32 outputs a signal representing completion of calculation of the energy difference values 

45 and the regular triangular mesh data received from the regular triangular mesh former 31 . to the control point remover 
33. 

The control point remover 33 stores the data received from the energy calculator 32 in the internal memory. Thus, 
the energy difference values corresponding to all the floating control points and the regular triangular mesh data are 
stored in the internal memory of the control point remover 33. The control poinl remover 33 compares the energy 

so difference* values stored in the internal memory with each other to determine a minimum energy difference value, and 
removes the data of the determined minimum energy difference value and the corresponding floating control point from 
the internal memory. That is. the control point remover 33 removes the floating control point corresponding to the 
minimum energy value as a non-control point. The control point remover 33 outputs the number of the remaining floating 
control points, the data of the region of support involving the removed floating control point, and the triangular mesh 

55 data representing the triangular meshes adjacent to the region of support, to a mesh reconstructor 34. 

The mesh reconstructor 34 divides the region of support from which the central floating control point is removed 
into the triangular meshes using the data which is received from the control point remover 33. The mesh reconstructor 
34 applies the above-described triangulation technique with respect to the region of support from which the central 
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floating control point has been removed. Therefore, the region of support from which the floating control point has been 
removed by the mesh reconstructor 34 is Divided into the new triangular meshes. The mesh reconstructor 34 judges 
whether the number of the remaining floating control points are not more than a predetermined threshold value. When 
f; the remaining floating control points is more than a predetermined threshold value, the mesh reconstructor 34 outputs 

s the data representing the newly obtained triangular meshes from the region of support from which the floating control 
■p point has been removed and the triangular.meshes adjacent to the new triangular meshes to an energy changer 35. 

The energy changer 35 which receives the data from the mesh reconstructor 34 calculates an energy difference value 
I with respect to the floating control points of the new triangular meshes. Each floating control point belonging to the 

I new triangular meshes has a corresponding regbn or support as in the region of support described referring to Figure 

10 5A. Therefore, when the new triangular meshes are determined as shown in Figure 5B : the energy changer 35 catcu- 
r lates new energy difference values corresponding to all the floating control points shown in Figure 58. The energy 

changer 35 outputs the data representing the calculated energy difference values and the corresponding Reeling control 
' point, to the control point remover 33. The control point remover 33 determines the minimum energy difference value 

using the data received from the energy changer 35 and the energy difference value stored in the internal memory and 
is removes the determined minimum energy difference value and the position data of the corresponding floating control 
point from the internal memory. 

The removal operation of the floating control point and the corresponding energy difference value by the control 
point remover 33. the mesh reconstructor 34 and the energy changer 35. is repeated, until the number of the remaining 
floating control points is identical to the predetermined threshold value. If it is determined that the number of the re- 
20 maining floating control points is identical to the predetermined threshold value, the mesh reconstructor 34 outputs the 
data of the floating control point and the triangular mesh data representing the triangular meshes both of which remain 
in the control point remover 33. to a position mover 40. 

An energy calculator 42 in the position mover 40 determines candidate control points each having a pixel position 
adjacent to the floating control point for each floating control point, using the floating control point data and the triangular 
2S mesh data which is received from the control point remover 33. An example of a candidate control point selected by 
the energy calculator 42 is shown in Figure 6. Preferably, the candidate control points are determined by four pixel 
positions or eight pixel positions which are just adjacent to the floating control point. Since determination of such 
candidate control points does not restrict the present invention, it is possible to further expand the range of selection 
of the candidate control points. The energy calculator 42 connects the determined candidate control points with the 
30 control points (i.e.. fixed and/or floating control points) adjacent to the floating control points, to thereby form new 
triangular meshes. Then, the energy calculator 42 calculates a third energy value corresponding to each candidate 
control point corresponding to each floating control point. 

The third energy value corresponding to each candidate control point is calculated according to the following equa- 
tion (2). 



40 

Hera S K represents a region of support corresponding to the floating control point K. and represents the number 
of pixels contained in the region of support S K . g(i r j) represents a pixel value of the pixel having a position fry^and g 
(i.j) represents an interpolated pixel value corresponding to the pixel value g(ij). The interpolated pixel value g(Lj) is 

-*s calculated using the pixel value g(i.j) £ind the interpolation weight values corresponding to the pixel value g(i,j). The 
interpolation weight values for the pixel having the position (ij) are uniquely determined using the pixel values of the 
control points contained in each triangular mesh to which a pixel having the position (Lj) belongs among the triangular 
meshes formed by each candidate control point and the control points located at the edge of the region o1 support S K . 
When the third energy values corresponding to aU the candidate control points involving each floating control point 

so are calculated, the energy calculator 42 outputs the. data representing each floating control point and the corresponding 
candidate control points, and the third energy values involving each floating control point and the candidate control 
points, to a position selector 44. The energy calculator 42 also outputs the triangular mesh data received from the 
mesh reconstructor 34 to the position selector 44. 

The position selector 44 determines a candidate control point or a floating control point corresponding to the rnin- 

55 imum third energy value among the received third energy values as an optimal floating control point. The positron 
mover 40 determines an optimal floating control point with respect to all the floating control points, and outputs the 
optimized irregular mesh data including the irregular meshes which are changed according to the determination of the 
optimal floating control poinls. the optimal fixed control points and the optimized floating control points, to a subsequent 



8 



EP 0 808 066 A2 

block (not shown). 

A modified embodiment for producing the irregular polygonal meshes will be described with reference lo Figure 7. 
The object image extractor 10 and the rectangular mesh former 20 shown in Figure 7 are the same as those of 
Figure 1 . An energy calculator 81 receives rectangular mesh data from the rectangular mesh former 20. The rectangular 
s mesh data represents an object image comprised of rectangular meshes. The energy calculator 81 calculates fourth 
energy values individually corresponding to the floating control points within the meshed object region using the input 
rectangular mesh data. The energy calculator 81 uses the above-described equation (1 ) to calculate the fourth energy 
value. In this case, S K represents a region of support comprised of rectangular meshes enclosing the K-th floating 
control point. The interpolation weight values to obtain the interpolated value g(i.j)are determined using the pixel values 

w of the control points belonging to the region of support. The energy calculator 81 calculates the fourth energy values 
involving each floating control point in the meshed object region, and stores the calculated fourth energy value in the 
internal memory (not shown) in the form corresponding to the floating control point. When the fourth energy values 
with respect to all the floating control points are completely calculated, the energy calculator 81 outputs a signal rep- 
resenting the completion of calculation of the fourth energy values to a control point remover 82. The control point 

is remover 82 removes the floating control point corresponding to the smallest fourth energy value among the fourth 
energy values stored in the energy calculator 81 from the internal memory. 

In this case, the newly formed mesh has a size different from an existing rectangular mesh. When the number of 
the floating control points remaining in the energy calculator 81 has a value larger than a predetermined threshold 
value, the control point remover 82 outputs an energy calculation instruction to the energy calculator 81 . The energy 

20 calculator 81 responding to the energy calculation instruction, newly calculates fourth energy values corresponding to 
the floating control points adjacent to removed floating control point and stores the calculated value. Also, the control 
point remover 82 removes the floating control point corresponding to the smallest fourth energy value among the fourth 
energy values stored in the energy calculator 71. If the floating control points are repeatedly removed by the energy 
calculator 81 and the control point remover 82. the initial regular rectangular meshes are deformed into irregular po- 

25 lygonal meshes whose size and shape differ from each other. If a signal processing is repeated for removing the floating 
control points, the number of floating control points remaining in the energy calculator 81 becomes the same as a 
predetermined threshold value. Here, the control point remover 82 reads the data representing the polygonal meshes 
stored in the energy calculator 81 and outputs the read polygonal mesh data to a position mover 90. The polygonal 
mesh data includes the data representing the fixed control points and the remaining floating control points and the 

30 image data involving the remaining polygonal meshes. 

When the position mover 90 processes the polygonal mesh data described with reference to Figure 7. the above- 
described equation (2) is changed to be adapted in the polygonal mesh structure. In the changed equation (2), the 
interpolation weight values for obtaining the interpolated value g(ij) are determined using the pixel values of each 
candidate control point and the control points which are the vertex points of each polygonal mesh formed by the control 

35 points located at the edge of the region of support S K . Since such a change is performed by the above-described 
equation (2) adapted for the triangular mesh structure to be adapted for the polygonal mesh structure, the detailed 
description thereof will be omitted. 

As described above, the image representation apparatus according to the present invention can process signals 
much more efficiently by performing a mesh representation with respect to an object extracted from the image than 

40 the case where a whole image is represented with meshes. Therefore, an efficiency of use of a memory required for 
the image mesh representation can be optimized and a system having a proper memory capacity can be additionally 
designed. 

The reader's attention is directed to all papers and documents which are filed concurrently with or previous to this 
specification in connection with this application and which are open to public inspection with this specification, and the 
•*5 contents of all such papers and documents are incorporated herein by reference. 

All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and/ 
or all of the steps of any method or process so disclosed, may be combined in any combination, except combinations 
where at least some of such features anoYor steps are mutually exclusive. 

Each feature disclosed in this specification (including any accompanying claims, abstract and drawings), may be 
50 replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. 
Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic series of equivalent 
or similar features. 

The invention is not restricted to the details of the foregoing embodiment(s). The invention extends to any novel 
one. or any novel combination, of the features, disclosed in this specification (including any accompanying claims. 
ss abstract and drawings), or to any novel one. or any novel combination, of the steps of any method or process so 
disclosed. 
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1. An objecl-oriemed image representation apparatus using irregular meshes comprising: 

s object image extraction means (10) for extracting image blocks containing an object contained in a whole 

image from the whole image represented by received image data 

regular mesh formation means (20) for receiving object image data from said object image extraction means 
and for generaling rectangular mesh data including fixed control points, floating control points and pixel daia 
jo from the object image, in which said fixed control points and said floating control points are vertex points 

belonging to an object image among vertex points of a plurality of the rectangular meshes which divide the 
whole image and said fixed control points are vertex points located at the corners of the object image: and 

irregular mesh formation means (30) for receiving the rectangular mesh data supplied from said regular mesh 
is formation means (20). and repeating an operation for removing said floating control point having image de- 

scriptiveness of the lowest degree and an operation for dividing a region of support from which the floating 
control point is removed into polygonal meshes, until the number of the remaining floating control points reach- 
es a predetermined threshold value. 

20 2. The object-oriented image representation apparatus according to claim t . wherein said object image extraction 
means (10) comprises: " 

block formation means (11) for dividing the whole image of the received image data into a plurality of image 
blocks each having a predetermined size: 

25 

block selection means (12) for selecting image blocks containing an object in the whole image among the 
plurality of the image blocks divided by said block formation means (11): and 

object image formation means (1 3) for merging the image btocks selected by said block selection means (12) 
so and outputting the merged image blocks as object image data. 

3 The objectoriented image representation apparatus according to claim 1 or 2. wherein said irregular mesh forma- 
tion means (30) performs a division operation for dividing the region of support from which the central floating 
control point has been removed into triangular meshes. 

4. The object-oriented image representation apparatus according to claim l t 2 or 3, wherein said irregular mesh 
formation means (30) comprises: 

regular triangular mesh formation means (31) for determining a diagonal direction having a large degree of 
40 data similarity of each rectangular mesh using the rectangular mesh data supplied from said regular mesh 

formation means (20). and forming regular triangular meshes by dividing the rectangular meshes along the 
determined diagonal direction: 

energy calculation means (32) for calculating energy difference values individually corresponding to the floating 
45 control points, using the pixel data of the regular triangular meshes formed by said regular triangular mesh 

formation means (31 ): . 

control point removal means (33) for determining the smallest energy difference value among the calculated 
energy difference values and removing the floating control point corresponding to the determined energy dif- 
so ference value as a non-floating control point: 

mesh reconstruction means (34). responsive to the floating control point removal operation of said control 
point removal means (33). for dividing the region of support from which the centre floating control point has 
been removed into the triangular meshes until the number of the floating control points reaches a predeter- 
55 mined threshold value: and 

energy changing means (35) for calculating the energy difference values individually corresponding to the 
floating control points contained in new triangular meshes using the new triangular meshes and the neigh- 
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bouring triangular meshes obtained by said mesh reconstruction means (34), anc^upplying information rep- 
resenting the calculated energy difference values and the corresponding floating control points to said control 
point removal means (33). 

The object-oriented image representation apparatus according to claim 4 t wherein the energy difference value 
corresponding to each floating control point K represents a difference between a first energy value and a second 
energy value, 

wherein the first energy value E(K) involving the floating control point K is represented as the following equation, 



/V A - ( i. /# ts S* 

wherein Sk represents a region of support corresponding to the floating control point K. represents the 
number of pixels contained in the region of support g(ij) represents a pixel value of the pixel having a 
position pj), and gpj) represents an interpolated pixel value corresponding to the pixel value gQ,j) wherein 
the interpolated pixel value gpj) is calculated using the pixel value gpj) and the interpolation weight values 
corresponding to the pixel value gpj). and the interpolation weight values for the pixel having the position p, 
j)are determined using the pixel values of the control points contained in each triangular mesh to which a pixel 
having the position pj) belongs within the region of support Sk before the central floating control point K is 
removed, and 

wherein the second energy value with respect to the floating control point K is calculated by the above equation 
using the interpolated pixel value determined by using the pixel values of the control points of the new triangular 
meshes, instead of the interpolated pixel value gpj) for the first energy value E(K). 

The object-oriented image representation apparatus according to claim 1 . wherein said irregular mesh formation 
means performs a division operation for dividing the region of support from which the centre floating control point 
has been removed into polygonal meshes 

The object-oriented image representation apparatus according to claim 6. wherein said irregular mesh formation 
means comprises: 

energy calculation means (81 ) for receiving and storing the rectangular mesh data supplied from said regular 
mesh formation means (20) and calculating and storing energy difference values individually corresponding 
to the floating control points using the stored rectangular mesh data: and 

control point removal means (82) for repeating a removal operation of a floating control point corresponding 
to the smallest energy value among the energy values stored in said energy calculation means until the number 
of the remaining floating control points reaches the predetermined threshold value. 

wherein said energy calculation means (81) newly calculates and stores the energy values corresponding to 
the floating control points contained in the region of support from which the centre floating control point has 
been removed. 

The object-oriented image representation apparatus according to claim 7, wherein said energy value is calculated 
by the following equation. 



in which represents a region of support corresponding to the floating control point K. Nk represents the 
number of pixels contained in the region of support Sk- gpj) represents a pixel value of the pixel having a 
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position and g(i t j) represents an interpolated pixel value corresponding to the pixel value g(i.j). 

wherein the interpolated pixel value g(i,j) is calculated using the pixel values g(i,j) and interpolation weight 
values corresponding to the pixel value gfi,j). and the interpolation weight values for the pixel having the po- 
5 sition (ij) are determined using the pixel values of the control points contained in the region of support S*. 

9. The object-oriented image representation apparatus according to claim 1 further comprising: 

energy calculation means (42) for calculating energy values involving each floating control point and a plurality 
w of candidate control points having a neighbouring pixel position in the region of support of each said floating 

control point, with respect to the floating control points of the polygonal meshes which finally remain in said 
irregular mesh formation means: and 

means for determining the smallest energy value among the energy values calculated by said energy calcu- 
15 lation means (42). and determining a control point corresponding to the determined energy value among each 

said floating control point and the said neighbouring candidate control points as an optimal floating control 
point corresponding to each said floating control point. 

10. The object-oriented image representation apparatus according to claim 9. wherein said energy value is calculated 
20 by the following equation, 



/V A - O.TTeS* 



25 



wherein S K represents a region of support corresponding to the floating control point K. N K represents the 
number of pixels contained in the region of support s* g(ij) represents a pixel value of the pixel having a 
so position (ij), and g(t.j) represents an interpolated pixel value corresponding to the pixel value g(i,j). 

wherein the interpolated pixel value g(ij) is calculated using the pixel values g(ij) and interpolation weight 
values corresponding to the pixel value g(ij), and the interpolation weight values for the pixel having the po- 
sition (ij) are determined using the pixel values of the control points of the triangular meshes to which the 
3S pixel having a position g($J) belongs among the triangular meshes formed by the floating control point or each 

said candidate control point and the control points located at the edge of the region of support Sk. 

11. A object-oriented image representation method using irregular meshes, comprising the steps of: 

40 (a) extracting image blocks containing an object contained in a whole image from the whole image: 

(b) generating rectangular mesh data including fixed control points, floating control points and pixel data from 
the image blocks extracted in step (a), in which said fixed control points and said floating control points are 
vertex points belonging to an object image among vertex points of a plurality of the rectangular meshes which 

45 divide the whole image and the fixed control points are vertex points located at the corners of the object image: 

and 

(c) repeating an operation tor removing said floating control point having image descriptiveness of the lowest 
degree and an operation tor dividing a region of support from which a centre floating control point is removed 

so into polygonal meshes, until the number ot remainingfloating control points reaches a predetermined threshold 

value, based on the rectangular mesh data of step (b). 

1 2. The object-oriented image representation method using irregular meshes according to claim 1 1 . wherein said step 
(a) comprises the sub-steps of: 

55 

(at) dividing the whole image into a plurality ol image blocks each having a first predetermined size: 

(a2) selecting image blocks containing an object in the whole image among the plurality of the image blocks 
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13. The object-oriented image representation method according to claim 11 or 12. wherein in said step (c). a division 
operation is performed (or dividing the region of support from which the centre floating control point has been 
removed into triangular meshes. 

14. The object-oriented image representation method according to claim 1 3, wherein said step (c) comprises the sub- 
steps of: 

(d) determining a diagonal direction, having a largedegree of data similarity of each rectangular mesh using 
the rectangular rhesh data of said step (b) and forming regular triangular meshes by dividing the rectangular 
meshes along the determined diagonal direction: 

(c2) calculating energy difference values individually corresponding to the floating control points using the 
pixel data of the regular triangular meshes formed in said sub-step (d ): 

(c3) determining the smallest energy difference value among the energy difference values calculated in said 
sub-step (c2) and removing the floating control point corresponding to the determined energy difference value 
as a non-floating control point: 

(c4) dividing the region of support from which the centre floating control point has been removed into the 
triangular meshes in response to the floating control point of said sub-step (c3) when the number of the floating 
control points does not reach a predetermined threshold value: and 

(c5) calculating the energy difference values individually corresponding to the floating control points contained 
in new triangular meshes using the new triangular meshes and the neighbouring triangular meshes obtained 
in said sub-step (c4). 

15. The object-oriented image representation method according to claim 14, wherein the energy difference value cor- 
responding to each floating control point K represents a difference between a first energy value and a second 
energy value. 

wherein the first energy value E(K) with respect to the floating control point K is represented as the following 
equation. 



wherein S K . represents a region of support corresponding to the floating control point K t N K represents the 
number of pixels contained in the region of support Sk, g(ij) represents a pixel value of the pixel having a 
position (Lj), and g(Lj) represents an interpolated pixel value corresponding to the pixel value g(Lj), wherein 
the interpolated pixel value gtf.j) is calculated using the pixel value g(Lj) and the interpolation weight values 
corresponding to the pixel value gfl.j). and the interpolation weight values for the pixel having the position (i. 
J) are determined using the pixel values of the control points contained in each triangular mesh to which a pixel 
having the position belongs within the region of support before the central floating control point K is 
removed, and 

wherein the second energy value with respect to the floating control point K is calculated by the above equation 
using the interpolated pixel value determined by using the pixel values of the control points of the new triangular 
meshes instead of the interpolated pixel value g(Kj). 

16. The object-oriented image representation method according to claim 11 . wherein in said step (c) ; a division oper- 
ation is performed for dividing the region of support from which the centre floating control point has been removed 
into rectangular meshes. 
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17. The object-oriented image representation method according to claim 1 6 : wherein said step (c) comprises the sub- 
steps of: 

(d) calculating and storing energy difference values individually corresponding to the floating control points 
using the stored rectangular mesh data of said step (b): and 

* (c2) removing the floating control point corresponding to the smallest energy value among the energy values 
stored in said sub-step (d ): 

(c3) calculating the energy values corresponding to the floating control points belonging to the region of support 
from which the centre floating control point has been removed: and 

(c4) repeating a removal operation of a floating control point and a calculation operation of the energy values 
corresponding to the floating control points belonging to the corresponding region of support until the number 
of the remaining floating control points reaches a predetermined threshold value. 

18. The object-oriented image representation method according to claim 17. wherein said energy value is calculated 
by the following equation. 



in which S K represents a region of support corresponding to the floating control point K. represents the 
number of pixels contained in the region of support Sk- g(r,j) represents a pixel value of the pixel having a 
position (ij), and g(ij) represents an interpolated pixel value corresponding to the pixel value g(i,j) t 

wherein the interpolated pixel value g(ij) is calculated using the pixel values g(ij) and interpolation weight 
values corresponding to the pixel value g(ij), and the interpolation weight values for the pixel having the po- 
sition (7./; are determined using the pixel values of the control points contained in the region of support S K . 

19. The object-oriented image representation method according to claim 11 further comprising the steps of: 

(d) calculating energy values involving each floating control point and a plurality of candidate control points 
having a neighbouring pixel position in the region of support of each said floating control point with respect 
to the respective floating control points of the polygonal meshes which finally remain in step (c): and * 

(e) determining the smallest energy value among the energy values calculatedby said step (d) and determining 
a control point corresponding to the determined energy value among each said floating control point and said 
neighbouring candidate control points as an optimal floating control point corresponding to each said floating 
control point. 

. 20. The object-oriented image representation method according to claim ig : wherein said energy value is calculated 
by the following equation. 



iV K H.>;*S € 



wherein. S K represents a region of support corresponding to the floating control point K. N K represents the 
number of pixels contained in the region of support g(i.j) represents a pixel value of the pixel having a 
position (Lj) and g(Lj) represents an interpolated pixel value corresponding to the pixel value g(i.j). 

wherein the interpolated pixel value g(ij) is calculated using the pixel values g(ij) and interpolation weight 
values corresponding to the pixel value gflj), and the interpolation weight values for the pixel having the po- 
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sition (ij) are determined using the pixel values of the control points of the triangular meshes to which the 
pixel having a position g(ij) belongs among the iriangular meshes formed by the floating control point or each 
said candidate control point and the control points located at the edge of the region of support S^. 
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(54) Object-oriented image representation method and apparatus using irregular meshes 



(57) An object-oriented image representation appa- 
ratus using irregular meshes includes an object image 
extractor (10) for extracting image blocks containing an 
object contained in a whole image from the whole image 
represented by received image data. A regular mesh 
former (20) generates rectangular mesh data including 
fixed control points, floating control points and pixel data 
from the extracted object image. The fixed control points 
and the floating control points are vertex points belong- 
ing to an object image among vertex points of a plurality 
of the rectangular meshes which divide the whole image 
and the fixed control points are vertex points located at 



the comers of the object image. An irregular mesh 
former (30) receives the rectangular mesh data and re- 
peats an operation for removing the floating control point 
having image descrtptrveness of the lowest degree and 
an operation for dividing a region of support from which 
the floating control point is removed into polygonal 
meshes, until the number of the remaining floating con- 
trol points reaches a predetermined threshold value. 
Thus, since a mesh representation with respect to an 
object instead of a whole image can be done, a much 
more efficient signal processing is possible compared 
with the mesh representation of the whole image. 
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